The present invention relates to an image processing apparatus for supplying image data to a plurality of processor elements, performing image processing in parallel by SIMD (single instruction multiple data stream) control of these plurality of processor elements, and performing contour enhancement for enhancing the contours of an image and a method of the same and to a data processing apparatus for filtering for example image data by an FIR filter and a method of the same.
Color signals of a plurality of formats such as RGB signals, YIQ signals, and YCrCb signals (below, a luminance signal Y will also be treated as a color signal) are processed in the fields of image signal processing, television signal processing, etc.
In order to correct an optical system in a video camera, correct nonlinearity in a display device (display), impart a special effect, and so on, it sometimes becomes necessary to perform nonlinear processing such as color correction and xcex3 (gamma) correction with respect to these various types of color signals.
In the past, nonlinear processing of these various types of color signals had been carried out by analog processing. However, when a color signal is processed analogly, nonuniformity occurs in the processing among components of color signals or a changes occur along with time due to repetition of processing, so the image is deteriorated. Accordingly, at the present time, nonlinear processing of color signals is generally carried out by digital processing.
As the method of nonlinear processing by digital processing, for example, there can be mentioned the exclusive logic circuit method which uses a delay circuit, multiplier circuit, adder circuit, and other logic circuits for realization of nonlinear input/output characteristics by break point approximation. However, the former method requires that a logic circuit be prepared for every processing, therefore lacks flexibility in the processing content. Accordingly, this method is rarely adopted.
Further, as another method of nonlinear processing by digital processing, mention may be made, for example, of a method of establishing correspondence between the values of input data and the values of output data via a memory. According to this method, the content of the nonlinear processing can be easily changed by just changing the storage content of the memory establishing correspondence between the input data and the output data. However, in the related art, no method for designating the processing content by a GUI (graphical user interface) had yet been established.
In addition, the result of the nonlinear processing had been checked by once recording the image data obtained by the processing on a VTR tape etc. and then reproducing and displaying the recorded image data, which was very troublesome.
Further, the apparatus for establishing correspondence between the input and output data using this method was usually configured for only color correction, xcex3 correction, and other nonlinear processing, therefore it was necessary to place other dedicated hardware in front or back of it to perform the other processing. Accordingly, even when establishing correspondence between input and output data to carry out color correction and y correction, in the end, in the same way as the method using logic circuits, it was necessary to prepare dedicated hardware to handle the other processing.
On the other hand, in order to flexibly perform various types of image processing, the method of using a DSP (digital signal processor) to process the image data by software can be considered. However, while a DSP is normally suitable for linear processing, it is not suitable for color correction, xcex3 correction, and other nonlinear processing, therefore there was only a few examples of utilization of a DSP for the nonlinear processing in the past.
Further, contour enhancement is currently used in televisions, video cameras, VTR apparatuses, image editing apparatuses, special effect apparatuses, etc. for industrial use in television broadcast stations etc. In the future, it expected to be actively utilized in the image processing apparatuses of the general consumers as well.
In the past, contour enhancement apparatuses for performing the contour enhancement were realized by dedicated hardware constituted by multipliers, adders, etc.
However, contour enhancement is realized by processing for detecting the contours of the image of the object and processing for enhancing the detected contour part, therefore the hardware of the contour enhancing apparatus ends up becoming large in size.
Further, once dedicated hardware for performing these processings is prepared, it is difficult to change the frequency characteristic of the high pass filters used when detecting the contour parts or the degree of enhancement of the contour parts etc.
Further, filtering by digital processing is used in a wide range of fields such as image processing and audio processing at present. In particular, in the field of image processing, it is indispensable for a band limitation, recording, editing and imparting of special effects for television signals etc. and has been used for a wide range of purposes.
In the past, as the filtering apparatus for performing filtering by digital processing, for example, use has been made of an FIR filter apparatus comprised of a multiplier, adder, etc. with specifications fixed by the hardware.
Further, the design of such a FIR filter apparatus required work for calculation for determining the filter coefficient satisfying the desired passing band characteristic and element band characteristic and work for actually preparing an FIR filter apparatus for performing the filtering using the filter coefficient obtained as a result of the calculation and using the same to verify the characteristics of the filtering by hardware or for using circuit simulator software to verify the characteristics of the filtering by software.
However, when using the method of preparing FIR filters of individual specifications to verify the characteristics by hardware, a long time is required for the preparation of the FIR filters, so the development period of the filter apparatus becomes long.
Further, when using the method of verifying the characteristics by software, the filter processing can not be simulated in real time, therefore it is not possible to verify the characteristics by viewing the image data obtained by actual filtering.
Further, not suitable method has yet been conceived as the method of evaluating the effect of the filtering of the image data of a moving picture.
Further, it has been known that the filtering by an FIR filter can be carried out by software by using an SIMD-controlled linear array type multiple parallel processor and that the desired characteristics can be realized by this, but in the past there had been no development apparatus for uniformly performing everything from the determination of the filtering characteristics (specifications) of the program for making an SIMD-controlled linear array multiple parallel processor control perform filtering by an FIR filter to the verification (evaluation) of the characteristics.
Further, the procedures from the determination of the specifications of the filtering program of the SIMD-controlled linear array multiple parallel processor to the evaluation are difficult. It would be convenient if it were possible to perform this series of work by operation using a GUI for example.
The present invention was made so as to solve the above problems and has as an object thereof to provide an image processing apparatus capable of performing nonlinear processing such as color correction on image data by using for example a DSP and a method of the same.
Further, another object of the present invention is to provide an image processing apparatus enabling free setting of the content of nonlinear processing such as color correction for every component of the color signals (Y, Cr, Cb, R, G. B, etc.) by using a GUI and in addition enabling quickly confirmation of the result of the color correction etc. on a GUI screen and a method of the same.
Further, still another object of the present invention is to provide an image processing apparatus enabling contour enhancement by software by using an SIMD-controlled linear array type multiple parallel processor and a method of the same.
Further, still another object of the present invention is to provide an image processing apparatus enabling contour enhancement by simple setting of the characteristic of the filtering when detecting the contour of the image of the object in the image data and characteristics of the nonlinear conversion for adjusting the degree of contour enhancement by for example a GUI and in addition enabling quick confirmation of the result of the processing and a method of the same.
Further, still another object of the present invention is to provide a data processing apparatus enabling filtering by software by using an SIMD-controlled linear array multiple parallel processor and in addition enabling uniform determination of the filtering characteristic to verification of the characteristic and a method of the same.
Further, still another object of the present invention is to provide a data processing apparatus enabling a reduction of the development period of a filtering apparatus and a method of the same.
An image processing apparatus according to the present invention comprises an input use image displaying means for displaying an input use image showing an input/output characteristic between an input image data and an output image data; a designation data receiving means for receiving designation data input in accordance with the displayed input use image and designating the input/output characteristic; an input/output characteristic extracting means for extracting the input/output characteristic from the received designation data; an input/output characteristic image displaying means for displaying the input/output characteristic image showing the extracted input/output characteristic; and an image data processing means for processing the input image data to generate the output image data so that the input image data and the output image data have a relationship indicated by the extracted input/output characteristic.
Preferably, the input image displaying means displays a graph of an initial value of the input/output characteristic; the designation data receiving means receives at least a first designation data for designating addition of a passing point of a curve of the displayed graph and the position of the passing point to be added on the graph, a second designation data for designating a change of the position of the added passing point and the position of the passing point to be changed after the change, and a third designation data for designating deletion of the added passing point; and the input image displaying means changes the graph of the input/output characteristic based on the received first designation data to third designation data.
Preferably, the apparatus comprises an input/output characteristic data generating means for generating input/output characteristic data for establishing correspondence between the input image data and the output image data according to the extracted input/output characteristic; and the image data processing means converts the value of the input image data to the value of the output image data by a memory mapping method based on the generated input/output characteristic data.
Preferably, the image data processing means processes the input image data based on a set program to generate the output image data and comprises a program generating means enabling the image data processing means to prepare a program for generating the output image data from the input image data based on the extracted input/output characteristic.
Preferably, the image data processing means comprises a SIMD-controlled linear array type multiple parallel processor.
The input use image displaying means for example first displays a graph of the initial value (y=x) of the input/output characteristic showing to output the value (x) of a component (R, G, B, Y, I, Q, etc.; input image data) of the color signal to be processed without change as the value (y) of the component (output image data) of the color signal obtained as the result of processing in a window of a GUI image for every component of the color signal.
When the user for example uses a mouse to designate the addition of a passing point of the curve of the displayed graph, a position of the passing point to be added on the graph, a change of the position of the added passing point, the position after change or the deletion of the added passing point with respect to each window of the components of the color signal of the GUI image, the designation data receiving means receives the designation data showing these designations for every component of the color signal.
The input/output characteristic extracting means extracts a function (break point approximation function) showing the input/output characteristic indicated by the received designation data by for example a break point approximation line for every component of the color signal.
The input/output characteristic image displaying means displays a graph showing a break point approximation function showing the extracted input/output characteristic in each window of the components of the color signal of the GUI screen at a point of time when the input of all designation data is terminated.
The input/output characteristic data generating means generates input/output characteristic data for establishing correspondence of the value of the input image data with the output image data according to the extracted input/output characteristic for every component of the color signal based on the extracted break point approximation function.
The image data processing means stores for example the input/output characteristic data and processes the input image data for every component of the color signal to generate the output image data using the memory mapping method where the input image data is used as the address input and the value of the input/output characteristic data stored at the address indicated by the value of the input image data is used as the output image data, and displays the same.
Further, for example, where an SIMD-controlled linear array type multiple parallel processor is used to process the input image data for every component of the color signal to generate the output image data, the program generating means prepares a program for realizing the extracted input/output characteristic and downloads the same to the processor.
Further, the image processing method according to the present invention comprises the steps of displaying an input use image showing an input/output characteristic between input image data and output image data; receiving designation data input in accordance with the displayed input use image and designating the input/output characteristic; extracting the input/output characteristic from the received designation data; displaying an input/output characteristic image showing the extracted input/output characteristic; processing the input image data to generate the output image data so that the input image data and the output image data have a relationship indicated by the extracted input/output characteristic; and displaying the generated output image data.
Preferably, the method displays a graph of the initial value of the input/output characteristic; receives at least a first designation data for designating an addition of a passing point of a curve of the displayed graph and the position of the passing point to be added on the graph, a second designation data for designating a change of the position of the added passing point and the position of the passing point to be changed after the change, and a third designation data for designating deletion of the added passing point and changes the graph of the input/output characteristic based on the received first designation data to third designation data.
Preferably, the method generates input/output characteristic data for establishing correspondence between the input image data and the output image data according to the extracted input/output characteristic and converts the value of the input image data to the value of the output image data by a memory mapping method based on the generated input/output characteristic data.
Preferably, the method processes the input image data based on a set program to generate the output image data, prepares a program for generating the output image data from the input image data based on the extracted input/output characteristic, and executes that generated program to process the input image data to generate the output image data.
Further, the image processing apparatus according to the present invention comprises a characteristic image displaying means for displaying a characteristic image showing a characteristic of contour enhancement with respect to image data input from an external portion; a characteristic receiving means for receiving the characteristic of contour enhancement in accordance with an operation with respect to the displayed characteristic image; a characteristic image changing means for changing the characteristic image showing the characteristic of contour enhancement in accordance with the received characteristic of contour enhancement; and a contour enhancement means for performing the contour enhancement with respect to the input image data based on the received characteristic of contour enhancement.
Preferably, the characteristic image displaying means displays characteristic images showing each of a characteristic of a first nonlinear conversion with respect to the image data input from the external portion, a characteristic of a second nonlinear processing, and a characteristic of filtering; the characteristic receiving means receives each of the characteristic of first nonlinear conversion, the characteristic of second nonlinear processing, and the characteristic of filtering in accordance with an operation with respect to the displayed characteristic image; the characteristic image changing means changes the characteristic images showing each of the characteristic of first nonlinear conversion, the characteristic of second nonlinear processing, and the characteristic of filtering in accordance with the received characteristic of first nonlinear conversion, the characteristic of second nonlinear processing, and the characteristic of filtering; and the contour enhancement means comprises a first nonlinear processing means for applying first nonlinear conversion with respect to the image data based on the received characteristic of first nonlinear conversion, a contour detecting means for performing filtering on the first nonlinear converted image data based on the received characteristic of filtering to detect the contour of the image in the image data and generate contour data showing the detected contour, a second nonlinear processing means for applying second nonlinear processing to the generated contour data based on the received characteristic of second nonlinear conversion, a time delaying means for imparting a time delay corresponding to the first nonlinear processing, the generation of the contour data, and the second nonlinear processing to the image data input from the external portion, and an adding means for adding the second nonlinear processed image data and the delayed image data.
Preferably, the apparatus further comprises a displaying means for displaying the contour enhanced image data.
Preferably, the apparatus further comprises a program preparing means for preparing a program to be executed by the contour enhancement means based on the received characteristic of contour enhancement; and the contour enhancement means executes the prepared program to perform the contour enhancement with respect to the input image data.
Preferably, the apparatus is characterized in that the contour enhancement means is a SIMD-controlled multiple parallel processor.
The image processing apparatus according to the present invention enables the user to set the characteristics of various processings in the contour enhancement for enhancing the contours of an image, that is, the characteristic of filtering by a high pass filter in contour detection, and to set the characteristics of the nonlinear conversion before or after the filtering by performing an operation with respect to a GUI image, performs the contour enhancement by software in accordance with these settings, and displays the result of the processing to provide the same to the user for confirmation.
The characteristic image displaying means displays, with respect to image data input from an external portion for contour enhancement, an image showing the characteristic of nonlinear conversion (first nonlinear conversion; level depend) for the component for enhancement of the contour of the image of the object in the image data and the characteristic of the nonlinear processing (second nonlinear conversion; clispining) for suppressing unnaturalness of the image due to over-enhancing of the detected contour part, for example, in a window of a GUI image in the form of a graph showing the value of the pixel data after the nonlinear processing with respect to pixel data of the input image.
Further, the characteristic image displaying means displays an image showing the frequency characteristic of the high pass filter used for the filtering, when detecting for example the contour of the image of an object, in a window of a GUI image in the form of a graph of the frequency response.
The user for example uses a mouse etc. for an operation for modifying the curves of the graphs of the characteristics of the nonlinear processing and the high pass filter in the windows of the GUI image so as to input the characteristics of the level depend, clispining, and filtering and uses a mouse etc. to push predetermined buttons in the GUI image to finally set these characteristics.
The characteristic receiving means receives the characteristics of the processings input as mentioned above when for example the user finally sets the characteristics of the processings.
The characteristic image changing means successively changes and displays the curves of the graphs in accordance with a modification operation during the period when for example the user performs an operation for modifying the curves of the graphs by a mouse etc. before finally setting the characteristics of the processings and shows them to the user.
By viewing the curves of the graphs changed by the characteristic image changing means, the user can obtain a general grasp of the characteristics of the processings.
The program preparing means prepares a program for controlling the operation of the contour enhancement means based on characteristics of the processings received by the characteristic receiving means so that each processing exhibits the received characteristics.
The contour enhancement means is for example an SIMD-controlled linear array type multiple parallel processor which executes the program prepared by the program preparing means to perform the level depend, clispining, and filtering and thereby performs the contour enhancement by the characteristics desired by the user.
That is, in the contour enhancement means, the first nonlinear processing means executes the program to performs level depend for enhancing the contour of image data input from the external portion.
The contour detecting means performs filtering by a high pass filter with respect to the contour enhanced image data, detects the contour part of the image of an object having a high frequency, and generates contour data showing the detected contour.
The second nonlinear processing means performs clispining for preventing the contour from being over-enhanced and becoming unnatural when the contour data generated by the contour detecting means is combined with the original image data.
The time delaying means delays the image data input from the external portion by exactly the time required for the above processings to match the timing with the clispined contour data.
The adding means adds the delayed image data and the clispined contour data to generate the contour enhanced image data of the image.
Further, the image processing method according to the present invention comprises the steps of displaying a characteristic image showing a characteristic of contour enhancement with respect to image data input from an external portion; receiving the characteristic of contour enhancement in accordance with an operation with respect to the displayed characteristic image; changing the characteristic image showing the characteristic of contour enhancement in accordance with the received characteristic of contour enhancement; and performing the contour enhancement with respect to the input image data based on the received characteristic of contour enhancement.
Preferably, the method comprises the steps of displaying characteristic images showing each of a characteristic of first nonlinear conversion with respect to the image data input from the external portion, a characteristic of second nonlinear processing, and a characteristic of filtering; receiving each of the characteristic of first nonlinear conversion, the characteristic of second nonlinear processing, and the characteristic of filtering in accordance with an operation with respect to the displayed characteristic images; changing each of the characteristic images showing the characteristic of first nonlinear conversion, the characteristic of second nonlinear processing, and the characteristic of filtering in accordance with the received characteristic of first nonlinear conversion, the characteristic of second nonlinear processing, and the characteristic of filtering; applying first nonlinear conversion with respect to the image data based on the received characteristic of first nonlinear conversion; performing filtering on the first nonlinear converted image data based on the received characteristic of filtering to detect a contour of the image in the image data; generating a contour data showing the detected contour; applying second nonlinear processing to the generated contour data based on the received characteristic of second nonlinear conversion; imparting a time delay corresponding to the first nonlinear processing, the generation of contour data, and the second nonlinear processing to the image data input from the external portion; and adding the second nonlinear processed image data and the delayed image data.
Preferably, the method displays the contour enhanced image data.
Preferably, the method prepares a program of the contour enhancement based on the received characteristic of contour enhancement and executes the prepared program to perform the contour enhancement on the input image data.
Preferably, the method prepares a parameter file based on the received characteristic of contour enhancement and executes the program of the contour enhancement referring to this parameter file to perform contour enhancement on the input image data.
Preferably, the method is characterized in that a SIMD-controlled multiple parallel processor executes the contour enhancement program.
Further, the data processing apparatus according to the present invention comprises a characteristic image displaying means for displaying a characteristic image showing a characteristic of filtering on data of a signal input from an external portion; a characteristic receiving means for receiving the characteristic of filtering in accordance with an operation with respect to the displayed characteristic image; a characteristic image changing means for changing the characteristic image showing the characteristic of filtering in accordance with the received characteristic of filtering; and a filtering means for performing the filtering on the input data based on the received characteristic of filtering.
Specifically, the data of the signal is image data; and the apparatus further comprises an image displaying means for displaying the filtered image data.
Preferably, the apparatus further comprises a filter circuit designing means for designing a filter circuit for performing the filtering on the input data by the received characteristic of filtering and describing the designed filter circuit by a predetermined hardware description language.
Preferably, the apparatus further comprises a program preparing means for preparing a program to be executed by the filtering means based on the received characteristic of filtering; and the filtering means executes the prepared program to perform the filtering with respect to the input data.
Specifically, the filtering means is an SIMD-format multiple parallel processor which performs the filtering by an FIR filter.
The data processing apparatus according to the present invention enables the user to set a filtering characteristic with respect to image, audio, or other data by performing an operation with respect to the GUI image, performs the filtering by software in accordance with these settings, and displays the result of the processing to provide the same to the user for his/her confirmation.
The characteristic image displaying means for example displays on a monitor a GUI image showing the frequency passing band, frequency blocking band, and other desired characteristics in the filtering of the image data in the form of for example a graph.
The user for example performs a modification operation on the curve of the graph in the GUI image by using a mouse etc. to input the filtering characteristic and further depresses a predetermined button in the GUI image by using the mouse etc. to finally set the desired characteristic.
The characteristic receiving means receives the input characteristic of filtering when for example the user finally sets the desired characteristic of filtering.
The characteristic image changing means for example successively changes and displays the curves of the graphs in accordance with the modification operation while the user performs the modification operation of the curves of the graphs by a mouse etc. before finally setting the filtering characteristic and shows the same to the user. By viewing the curves of the graphs changed by the characteristic image changing means, the user can obtain a general grasp of the characteristic of filtering.
The program preparing means calculates the filter coefficient of the FIR filter based on the filtering characteristic received by the characteristic receiving means so as to show for example the received characteristic and uses the calculated filter coefficient to prepare the program for filtering to be executed by the filtering means.
The filtering means is for example an SIMD-controlled linear array type multiple parallel processor which executes the program prepared by the program preparing means to perform the filtering on the image data by the characteristic desired by the user.
The filter circuit designing means for example designs the circuit of the FIR filter for filtering the image data by hardware by the characteristic desired by the user and generates a description of the content of the designed filter circuit by the HDL (hardware description language) or other hardware description language.
Further, the data processing method according to the present invention comprises the steps of displaying a characteristic image showing a characteristic of filtering on data of a signal input from an external portion; receiving the characteristic of filtering in accordance with an operation with respect to the displayed characteristic image; changing the characteristic image showing the characteristic of filtering in accordance with the received characteristic of filtering; and performing the filtering on the input data based on the received characteristic of filtering.
Specifically, the data of a signal is image data; and further the method displays the filtered image data.
Preferably, further, the method designs a filter circuit for performing the filtering on the input data by the received characteristic of filtering and describes the designed filter circuit by a predetermined hardware description language.
Preferably, further, the method prepares a program for realizing the filtering based on the received characteristic of filtering and executes the prepared program to perform the filtering on the input data.
Specifically, an SIMD-format multiple parallel processor performs the filtering by an FIR filter.